Skip to content

ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation#6323

Draft
clebertsuconic wants to merge 2 commits into
apache:mainfrom
clebertsuconic:message-size-amqp
Draft

ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation#6323
clebertsuconic wants to merge 2 commits into
apache:mainfrom
clebertsuconic:message-size-amqp

Conversation

@clebertsuconic

Copy link
Copy Markdown
Contributor

No description provided.

@clebertsuconic clebertsuconic force-pushed the message-size-amqp branch 8 times, most recently from f70cda5 to be90ac4 Compare March 29, 2026 21:44
@clebertsuconic clebertsuconic changed the title ARTEMIS-TBD Improving memory estimates on AMQP ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation and make it static Mar 29, 2026
@clebertsuconic clebertsuconic force-pushed the message-size-amqp branch 9 times, most recently from 0611d5a to 03b7176 Compare March 30, 2026 22:51
@clebertsuconic clebertsuconic marked this pull request as ready for review March 30, 2026 22:51
@clebertsuconic clebertsuconic force-pushed the message-size-amqp branch 8 times, most recently from 94efe35 to 70beb3a Compare April 1, 2026 13:07
@clebertsuconic clebertsuconic changed the title ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation and make it static ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation Apr 1, 2026
@tabish121 tabish121 self-requested a review April 1, 2026 14:14

@tabish121 tabish121 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, nice simplification of some of the message handling

@clebertsuconic clebertsuconic force-pushed the message-size-amqp branch 3 times, most recently from 09bb86b to 540ad20 Compare April 8, 2026 00:58
@clebertsuconic

Copy link
Copy Markdown
Contributor Author

The reason I had to add a V4 is because of Paging:

On this part here, I encode the number of queues and the queueIDs:

int queueIDsSize = buffer.readInt();
queueIDs = new long[queueIDsSize];
for (int i = 0; i < queueIDsSize; i++) {
queueIDs[i] = buffer.readLong();
}

At the point of the encoder I don't have any reference to the number of bytes used by its own decoder.

I'm adding one on V4 now, if in the future anything else is added, we will stop reading at that marker.

@clebertsuconic clebertsuconic force-pushed the message-size-amqp branch 4 times, most recently from 1cb4403 to fa1128e Compare April 8, 2026 01:26
@clebertsuconic

Copy link
Copy Markdown
Contributor Author

I intend to squash the second commit on the first as soon some review is done on the new persister.

@clebertsuconic

Copy link
Copy Markdown
Contributor Author

I'm setting it as ready to review. but this commit should be squashed before merged.

@clebertsuconic

Copy link
Copy Markdown
Contributor Author

I needed to add versioning to AMQPLargeMessagePersister as well.

I could choose to skip memory calculations on large message... however I will prefer to keep it the same way as StandardMessage.

this is also better for future additions

@clebertsuconic

Copy link
Copy Markdown
Contributor Author

I added a commit addressing the comments, let me know when I can squash it please

@clebertsuconic

Copy link
Copy Markdown
Contributor Author

I added another commit to be squashed after review

@clebertsuconic

Copy link
Copy Markdown
Contributor Author

when I worked on this I wasn't planning to rewrite persistence for AMQP messages.

I will work on something for those persisters.

@clebertsuconic

Copy link
Copy Markdown
Contributor Author

I am now using a MacCodec instead of a streamed bytes. I am not using a Hashmap to store records in between, but there's a codec that will store it like it was a hashMap.

I also added a clustering compatibility tests since the persistence is used with clustering.

@gemmellr gemmellr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gave it a skim over time, havent closely looked at most of it, but still had various comments so here they are. Most important one being, I do not think this should be in 2.54.0 at this stage.

Comment on lines +36 to +41
private static final ThreadLocal<AMQPMessageMapCodec> threadLocal = ThreadLocal.withInitial(AMQPMessageMapCodec::new);

public static AMQPMessageMapCodec getInstance() {
AMQPMessageMapCodec reader = threadLocal.get();
return reader;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly over-thinking it, but the this got me wondering if rather than a subclass, this would work nicer as a method that took a processor / callback-handler style argument. The persister would then just need 1 of those while it worked during startup and then it would go away when done, unlike this. You could also then separately test the codec class (current parent) and the processor/callback fully that way rather than e.g creating a test subclass to test the parent bits, and not really directly testing this class at all [it seems so far].

The large and regular persister could still use the same processor and then just toggle a flag to indicate which behaviour to give.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gemmellr I can do that. although I'm using a thread local now because of the state I needed to store before creating a message.

There's one changed I wanted to make on the constructors for AMQPStandardMessage and AMQPLargeMessage, that I could keep the implementation stateless, and just post data directly on the message. then I wouldn't need a ThreadLocal and I could just work with the message directly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure about a callback. I tried that before I sent the PR and I didn't like it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gemmellr I played with extension versus plugin.. it doesn't change much in the code.

We will still have a single ThreadLocal... I will keep it as extension for now

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont recall any need for a ThreadLocal in the approach my comment wondered about, pretty sure removing that that was what made me think of it. Unclear what would you be storing in it with the state in the processor/callback and the rest being static methods or maybe a singleton. It also seemed like it would fairly significantly change to the code and the [direct] testing, particularly of the current subclass.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main issue is that I need to know the messageFormat before creating an instance of AMQPStandardMessage or AMQPLargeMessage.

I have relaxed this a bit now and won't require any threadLocals any longer and will just parse it directly onto the mesasge object. (I will be push the commit shortly)

@clebertsuconic clebertsuconic Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gemmellr I tried decoding directly into the message but it made things worse.

Historically we've always created the message first, then applied reload. Inverting this for AMQPStandardMessage seemed doable but required a lot of new code. For AMQPLargeMessage it's not feasible at all—the messageID is tied to a file, and adding reload methods just creates more problems than it solves.

I'd keep the current approach with ThreadLocal state. If you want the codec stateless, we'd still need a ThreadLocal or temporary instance to hold the data before message instantiation anyway.

@clebertsuconic

Copy link
Copy Markdown
Contributor Author

@gemmellr I am marking this as draft because of the versioning.. whenever we decide to merge it I will change it and make it ready to commit

- making it immutable to avoid races after updates like we had in the past
- avoiding scanning after reloading by persisting extra data on storage
Comment on lines +36 to +41
private static final ThreadLocal<AMQPMessageMapCodec> threadLocal = ThreadLocal.withInitial(AMQPMessageMapCodec::new);

public static AMQPMessageMapCodec getInstance() {
AMQPMessageMapCodec reader = threadLocal.get();
return reader;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont recall any need for a ThreadLocal in the approach my comment wondered about, pretty sure removing that that was what made me think of it. Unclear what would you be storing in it with the state in the processor/callback and the rest being static methods or maybe a singleton. It also seemed like it would fairly significantly change to the code and the [direct] testing, particularly of the current subclass.

@clebertsuconic

Copy link
Copy Markdown
Contributor Author

I have added a commit addressing some of the comments.. I still need to finish this , but it will be tomorrow. for now I have made this a draft again.

@clebertsuconic

Copy link
Copy Markdown
Contributor Author

Only point I have open now is the discussion on the ThreadLocal for AMQPMessageMetadataPersister.

Making it stateless will require intermediate state somewhere else... or a bigger change in the order messages are created versus reloaded. (I want to avoid such inversion at this point)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants